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Abstract 

In  today’s  fiscal  environment,  federal  programs  must  be  postured  to  conduct  on-going 
tradeoff  analyses  to  stay  affordable  as  budgets  are  reduced  and  capabilities  change  or 
become  more  challenging  to  implement.  This  research  focuses  on  recommended  practices 
for  conducting  economic  resource-constrained  tradeoff  analyses.  The  goal  is  to  offer 
guidance  to  programs  in  making  cost-effective  affordability  decisions  that  keep  the  program 
within  its  budget,  or  to  find  economic  efficiencies  if  the  program  is  currently  affordable  and 
within  its  budget. 

Background 

Over  the  years,  there  have  been  major  efforts  within  the  federal  government  to 
reduce  the  cost  of  acquiring  systems.  The  Government  Accountability  Office  has  shown 
repeated  problems  in  meeting  program  milestones  and  keeping  programs  within  cost  and 
schedule  requirements.  Since  2010,  the  Department  of  Defense  (DoD)  has  issued  three 
versions  of  Better  Buying  Power  for  the  DoD  acquisition  community,  emphasizing  the  need 
for  “affordability”  (Carter,  2010;  Kendall,  2012,  2015). 

In  response  to  these  problems,  the  MITRE  Corporation  conducted  an  internal 
research  project  resulting  in  the  Affordability  Engineering  Framework  (AEF).  The  AEF, 
shown  at  a  high-level  in  Figure  1,  provides  a  structured  framework  with  approaches  and 
tools  to  address  program  affordability  challenges  over  the  life  cycle  (MITRE  Corporation, 
2012). 
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Figure  1.  AEF  Framework 

Step  1  of  the  AEF  is  to  identify  and  assess  potential  risks  to  program  affordability, 
and  to  initiate  actions  to  mitigate  those  risks.  Step  2  is  to  examine  the  sufficiency  of  the 
program  baseline  and  corresponding  cost  and  schedule  estimates,  then  to  compare  this 
cost  estimate  with  the  budget  profile  to  evaluate  life  cycle  affordability.  Step  3  is  to  conduct  a 
tradeoff  analysis  of  courses  of  action  for  the  purpose  of  making  the  program  affordable  or,  if 
the  program  is  currently  affordable,  to  explore  opportunities  to  improve  program  efficiencies. 
Finally,  the  objective  of  Step  4  is  to  help  the  decision-maker  select  the  appropriate  course(s) 
of  action. 

The  research  described  in  this  summary  is  intended  to  facilitate  Step  3  of  the  AEF. 
Specifically,  the  purpose  of  our  research  is  to  (part  1)  gain  an  understanding  of  how 
government  program  offices  currently  conduct  tradeoff  analyses  and  (part  2)  develop  a 
guidance  document  and  a  software  tool  to  help  them  with  this  process.  This  summary 
describes  shortfalls  in  current  practice  that  came  to  our  attention  during  part  1  and 
recommendations  for  correcting  these  shortfalls.  These  recommendations  will  influence  part 
2,  which  is  now  under  way. 

Approach  and  Findings 

To  better  understand  how  program  offices  deal  with  affordability  challenges  and 
tradeoff  analyses,  we  conducted  interviews  with  MITRE  staff  supporting  19  government 
sponsors.  The  main  topics  of  the  interviews  were  to  understand  the  following:  what 
affordability  tradeoff  analyses  programs  typically  conduct  and  what  decisions  are  supported 
by  the  results  of  the  analyses;  what  factors  (inputs)  are  considered  in  conducting  tradeoff 
analyses  and  how  are  they  considered;  and  what  resources  (people,  tools,  time)  are 
available  to  conduct  these  analyses.  The  information  we  gained  from  conducting  these 
interviews  will  assist  us  in  part  2  of  our  research  (developing  a  guidance  document  and 
tool).  Meanwhile,  this  summary  paper  deals  with  shortfalls  in  current  practice  that  came  to 
our  attention  in  the  course  of  the  interviews.  These  shortfalls  were  in  three  areas:  measuring 
benefit,  combining  metrics,  and  assessing  risk  in  tradeoff  analyses. 

Measuring  Benefit 

In  many  studies,  the  only  measure  of  benefit  was  a  technical  performance  measure, 
such  as  the  speed  of  an  aircraft.  No  attempt  was  made  to  connect  this  to  a  measure  of 
effectiveness,  which  expresses  how  well  the  system  carries  out  its  operational  mission.  This 
approach  leads  to  a  “more  is  better”  outlook.  It  does  not  provide  insight  into  how  much 
operational  value  is  diminished  when  a  lower  cost  alternative  is  selected  and  whether  this  is 
acceptable.  In  studies  that  used  metrics  that  were  not  easy  to  express  in  scientific  units, 
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such  as  “the  ability  to  conduct  close  air  support,”  the  system  that  was  adopted  for  scoring 
was  often  not  carefully  constructed.  For  example,  numerical  scores  were  not  given  clear 
interpretations. 

The  authors  recommend  adherence  to  established  decision  analysis  methods  for 
rating  value  or  utility.  For  further  information  on  these  methods,  see  Von  Winterfeldt  and 
Edwards’  (1986)  Decision  Analysis  and  Behavioral  Research,  Chapter  7. 

Combining  Metrics 

In  studies  that  combined  the  scores  of  several  metrics,  weighted  averages  were 
almost  always  used.  Although  the  weighted  average  is  known  to  be  the  correct  function  to 
use  when  certain  independence  conditions  hold  (see,  for  example,  Kirkwood,  1997,  p.  243), 
there  are  cases  in  which  such  conditions  do  not  hold,  as  is  illustrated  by  a  classic  study  of 
the  Mexico  City  airport  (Keeney  &  Raiffa,  1993,  Chapter  8).  The  inappropriate  choice  of  a 
function  can  lead  to  misleading  assessments  of  overall  benefit.  For  example,  an  alternative 
that  improves  the  overall  benefit  score  may  be  one  that  is  improving  metrics  that  are  already 
at  an  acceptable  level,  while  leaving  other  metrics  below  acceptable  levels. 

To  ensure  that  tradeoffs  are  represented  realistically,  the  authors  recommend  that 
analysts  be  aware  of  the  existence  of  functions  other  than  averages  that  can  be  used  to 
combine  metrics.  Some  examples  are  the  Multiplicative  Utility  Function  (Keeney  &  Raiffa, 
1993),  the  exponential  average  (Schmidt,  2015),  and  the  max-average  (Lamar,  2009).  We 
are  continuing  research  to  investigate  methods  for  making  these  concepts  more 
understandable  and  useable  for  program  tradeoffs. 

Assessing  Risk  in  Tradeoff  Analyses 

Risk  was  often  not  considered,  or  was  considered  improperly,  in  affordability  tradeoff 
analyses.  For  example,  in  some  studies  risk  was  assessed  for  one  candidate  system  but  not 
for  another.  In  other  studies,  only  one  type  of  risk  (e.g.,  schedule  risk)  was  considered,  while 
other  types  (e.g.,  cost,  technical  maturity,  interoperability,  and  statutory/regulatory)  were 
ignored. 

Our  recommendation  is  to  consider  what  we  call  the  execution  risk  framework 
(Henry,  2011).  This  method  evaluates  each  alternative  across  a  number  of  risk  sources  or 
categories.  For  each  alternative,  the  risk  for  each  category  is  assessed  using  a  utility-like 
scale.  Once  an  assessment  is  made  for  each  category,  these  scores  can  be  combined  using 
a  variety  of  methods,  including  the  max-average  (Lamar,  2009)  or  exponential  average 
(Schmidt,  2015).  Risk  scores  can  then  be  used  to  calculate  risk-adjusted  benefit.  In  addition, 
understanding  where  there  is  risk  for  a  given  alternative  guides  the  formulation  of  new  risk- 
reduction  alternatives,  which  include  risk  mitigation  activities  and  costs  for  those  activities. 

Next  Steps 

The  next  step  in  this  research  (part  2)  will  be  to  construct  a  guidebook  on 
recommended  practices  and  a  software  tool  to  help  program  offices  make  analytically-driven 
tradeoff  decisions.  This  research  will  result  in  a  simple-to-use  tool  enabling  programs  to 
conduct  affordability  tradeoff  analyses  on  a  regular  basis.  Although  this  study  focused  on 
DoD  program  offices,  our  intent  is  that  all  federal  agencies  will  gain  from  the  findings  of  this 
research  and  the  products  that  will  become  available. 
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Introduction 


■  In  today’s  fiscal  environment,  federal  programs  must  be 
postured  to  conduct  on-going  tradeoff  analyses  to  stay 
affordable 

-  As  budgets  are  reduced,  need  to  re-scope  wisely 

-  As  capabilities  change  or  become  more  challenging  to  implement, 
need  to  stay  within  the  budget  in  the  life-cycle 

■  Our  research  focuses  on  recommended  practices  for 
conducting  economic  resource-constrained  tradeoffs 

-  Part  1  was  designed  to  understand  how  government  offices 
currently  conduct  these  affordability  tradeoffs,  where  there  is  need 
for  improvement 

-  Part  2  will  be  to  develop  guidance  for  recommended  approaches 
and  develop  a  software  tool  to  help  programs  implement  these 
recommended  approaches 

■  This  paper  describes  our  findings  for  Part  1 


©2016  The  MITRE  Corporation.  All  rights  reserved. 


MITRE 


Want  versus  Need  -  Fuel  Efficiency* 


Tesla  Model  S  accelerates  from  0  to  60  priUs  Model  2  fuel  economy  (MPG) 

mph  in  as  little  as  2.8  seconds.  54 . 58  city  |  50  -  53  highway 

Trade-Off  Analysis 

Costs:  $80,000  $22,000 


Risks:  Speed  Few 

Benefits:  Neighbor  envy  MPG  bragging  rights 


*Sources:  Wikipedia  and  auto  manufacturer  websites 
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Background 


■  Over  the  years  there  have  been  major  efforts  within  the 
federal  government  to  reduce  the  cost  of  acquiring  systems 

■  The  Government  Accounting  Office  (GAO)  has  shown 
repeated  problems  in  meeting  program  milestones  and 
keeping  programs  within  cost  and  schedule  requirements 

■  Since  2010,  the  Department  of  Defense  (DoD)  has  issued 
three  versions  of  “Better  Buying  Power”  for  the  DoD 
acquisition  community,  emphasizing  the  need  for 
“affordability.”  [1],  [4],  [5]  (See  slide  12  for  references) 

■  In  response  to  these  challenges,  The  MITRE  Corporation 
developed  the  Affordability  Engineering  Framework  (AEF) 
[9],  which  provides  a  structured  framework  with  approaches 
and  tools  to  address  program  affordability  challenges  over 
the  life-cycle 
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The  Affordability  Engineering  Framework 
(AEF) 


■  The  MITRE-developed  AEF 
framework  [9]  is  a  continuous 
process  that  consists  of  four 
steps: 

-  Step  1  guides  programs  through 
a  self-assessment  of  risks  to 
affordability  based  on  where  they 
are  in  the  life-cycle 

-  Step  2  is  a  comparison  of  the 
Program  Office  Estimate  (POE) 
with  the  budget 

-  Step  3  provides  guidance  for 
conducting  tradeoffs  to  ensure 
affordability  and/or  improve 
efficiencies  and  effectiveness 

-  Step  4  is  the  selection  of  the  best 
course  of  action  and 
summarizing  for  decision-makers 


Step  1 

Conduct  Affordability 
Risk  Assessment 

| 

f 

Step  2 

Conduct  Affordability 
Evaluation 

Step  4 

Assess  Courses  of 
Action  and  Make 
Recommendation 

1 

Step  3 

Conduct  Tradeoff 
Analysis 

The  research  in  this  paper  is 
intended  to  faciiitate  Step  3  of  the 

AEF 
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Part  1  Approach:  Program  Interviews 


■  Part  1  of  this  research  consisted  of  19  interviews  with 
government  sponsors 

■  Main  topics  of  the  interviews  were: 

-  What  affordability  tradeoff  analyses  are  typically  conducted  and 
what  decisions  are  supported  by  the  analyses 

-  What  factors  (inputs)  are  considered  and  how 

-  What  resources  (time,  people,  tools)  are  available  to  conduct  these 
analyses 

■  The  information  gained  from  these  interviews  will  help  in  the 
development  of  the  guidance  document  and  tool 

■  Of  the  findings,  three  were  prevalent: 

-  Poor  practice  in  choosing  benefits  metrics  and  measuring  benefits 

-  Combining  metrics  limited  to  linear  methods 

-  Risk  not  considered  or  considered  incorrectly 
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Finding  1:  Benefits  Metrics  and  Measurement 


■  Problem:  Often  the  measure  of  benefit  was  technical 
performance  (e.g.,  the  speed  of  an  aircraft)  and  the  measures 
were  not  linked  to  extent  of  achieving  goals/mission/objectives 

-  Leads  to  a  “more  is  better”  outlook 

-  Does  not  allow  for  exploring  affordability  trades 

■  What  is  the  impact  to  goals/missions/objectives  of  pursuing  a  lower 
cost,  lower  performing  alternative?  Is  this  impact  acceptable? 

■  Problem:  Numerical  scores  for  benefit  were  not  given  clear 
interpretations 

■  Solution:  Ensure  that  technical  measures  are  linked  to  mission- 
level  metrics,  and  adhere  to  established  decision  analysis 
methods  for  rating  value  or  utility  [11,  Chapter  7] 
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Finding  2:  Combining  Benefits  Metrics 


■  In  studies  that  combined  the  scores  of  several  metrics,  weighted 
averages  were  almost  always  used 

■  Although  the  weighted  average  is  the  correct  function  to  use 
when  certain  independence  conditions  hold,  there  are  cases 
where  these  conditions  do  not  hold  [7],  [3] 

-  E.g.,  a  classic  study  of  Mexico  City  Airport  found  that  weighted 
average  was  not  appropriate  [3,  Chapter  8] 

■  The  inappropriate  choice  of  a  function  can  lead  to  misleading 
assessments  of  overall  benefit 

-  E.g.,  an  alternative  that  improves  the  overall  benefit  score  may  be 
one  that  is  improving  metrics  already  at  an  acceptable  level,  while 
leaving  other  metrics  below  acceptable  levels 

■  Solution:  Analysts  should  be  aware  that  the  weighted  average  is 
not  always  appropriate  and  be  aware  of  alternative  functions  for 
combining  benefits  metrics 

-  E.g.,  exponential  average  and  max-average  [8],  [10] 
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Finding  3:  Alternative  Risk  Assessments 


■  Problem:  The  risk  of  achieving  the  measured  benefits  for  the 
estimated  costs  is  not  often  considered,  or  considered  improperly 

-  Must  consider  risk  from  all  sources  (e.g.,  technical  maturity, 
interoperability,  statutory/regulatory),  not  just  the  standard  sources  of 
cost,  performance  and  schedule 

■  Solution:  Consider  what  we  refer  to  as  the  “execution  risk 
framework”  [2] 

-  Evaluate  risk  for  each  alternative  across  a  number  of  risk  sources 
using  a  utility-like  scale 

-  Scores  can  be  combined  using  methods  discussed  under  Finding  2 

-  Understanding  the  risk  of  an  alternative  can  lead  to  new 
alternatives  which  contain  mitigation  efforts  and  the  cost  of  those 
mitigations 
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Summary  and  Next  Steps 


■  Part  1  of  our  research  efforts  found  a  few  key  areas 
where  improvement  is  needed  as  program  offices  are 
challenged  to  understand  how  to  keep  programs 
affordable 

■  The  next  step  (part  2)  will  be  to  construct  a 
guidebook  on  recommended  practices,  leveraging 
work  done  for  the  MITRE  AEF,  and  development  of  a 
software  tool  to  help  programs  make  analytically- 
driven  tradeoff  decisions  on  a  regular  basis 

■  The  intent  is  that  all  federal  agencies,  DoD  and 
civilian,  will  gain  from  findings  on  this  research  and 
the  products  that  will  become  available 
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Want  versus  We  Think  We  Need  -  Stealth* 


Lockheed  Martin  F-35  Lightning  II 

F-35A  Cost:  $85M,  full  production 
in  FY2018 


Sometimes  you  just  need  a  nice  ride 
*Sources:  Wikipedia 
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The  F-117 
Nighthawk 

stealth 

attack 

aircraft 


The  B-2  Spirit 

strategic 

stealth 

bomber 


The  F-22  Raptor 

fifth  generation 
stealth  air 
superiority  fighte 


F-22  $150M/Unit 
F-117  $111  M/Unit 
B-2  $737M/Unit 
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